<!DOCTYPE html>
<html id="docs" lang="en" class="">
	<head>
	<meta charset="utf-8">
<title>Troubleshoot Clusters - Kubernetes</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="shortcut icon" type="image/png" href="../../../images/favicon.png">
<link rel="stylesheet" type="text/css" href="../../../css/base_fonts.css">
<link rel="stylesheet" type="text/css" href="../../../css/styles.css">
<link rel="stylesheet" type="text/css" href="https://code.jquery.com/ui/1.12.1/themes/smoothness/jquery-ui.css">
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/sweetalert/1.1.3/sweetalert.min.css">
<link rel="stylesheet" type="text/css" href="../../../css/callouts.css">
<link rel="stylesheet" type="text/css" href="../../../css/custom-jekyll/tags.css">




<meta name="description" content="Troubleshoot Clusters" />
<meta property="og:description" content="Troubleshoot Clusters" />

<meta property="og:url" content="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/" />
<meta property="og:title" content="Troubleshoot Clusters - Kubernetes" />

<script
src="https://code.jquery.com/jquery-3.2.1.min.js"
integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4="
crossorigin="anonymous"></script>
<script
src="https://code.jquery.com/ui/1.12.1/jquery-ui.min.js"
integrity="sha256-VazP97ZCwtekAsvgPBSUwPFKdrwD3unUfSGVYrahUqU="
crossorigin="anonymous"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/sweetalert/1.1.3/sweetalert.min.js"></script>
<script src="../../../js/script.js"></script>
<script src="../../../js/custom-jekyll/tags.js"></script>


	</head>
	<body>
		<div id="cellophane" onclick="kub.toggleMenu()"></div>

<header>
    <a href="../../../index.html" class="logo"></a>

    <div class="nav-buttons" data-auto-burger="primary">
        <ul class="global-nav">
            
            
            <li><a href="../../home.1">Documentation</a></li>
            
            <li><a href="../../../blog/index.html">Blog</a></li>
            
            <li><a href="../../../partners/index.html">Partners</a></li>
            
            <li><a href="../../../community/index.html">Community</a></li>
            
            <li><a href="../../../case-studies/index.html">Case Studies</a></li>
            
            
             <li>
                <a href="../cluster-troubleshooting.1#">
                    English <span class="ui-icon ui-icon-carat-1-s"></span>
                </a>
                <ul>
                
                    <li><a href="../../../zh/index.html">中文 Chinese</a></li>
                
                    <li><a href="../../../ko/index.html">한국어 Korean</a></li>
                
                </ul>
            </li>
         
            <li>
                <a href="../cluster-troubleshooting.1#">
                    v1.11 <span class="ui-icon ui-icon-carat-1-s"></span>
                </a>
                <ul>
                
                    <li><a href="https://kubernetes.io">v1.12</a></li>
                
                    <li><a href="../../../index.html">v1.11</a></li>
                
                    <li><a href="https://v1-10.docs.kubernetes.io">v1.10</a></li>
                
                    <li><a href="https://v1-9.docs.kubernetes.io">v1.9</a></li>
                
                </ul>
            </li>
        </ul>
        
        <a href="../../tutorials/kubernetes-basics/index.html" class="button" id="tryKubernetes" data-auto-burger-exclude>Try Kubernetes</a>
        <button id="hamburger" onclick="kub.toggleMenu()" data-auto-burger-exclude><div></div></button>
    </div>

    <nav id="mainNav">
        <main data-auto-burger="primary">
        <div class="nav-box">
            <h3><a href="../../tutorials/stateless-application/hello-minikube/index.html">Get Started</a></h3>
            <p>Ready to get your hands dirty? Build a simple Kubernetes cluster that runs "Hello World" for Node.js.</p>
        </div>
        <div class="nav-box">
            <h3><a href="../../home.1">Documentation</a></h3>
            <p>Learn how to use Kubernetes with the use of walkthroughs, samples, and reference documentation. You can even <a href="../../../editdocs/index.html" data-auto-burger-exclude>help contribute to the docs</a>!</p>
        </div>
        <div class="nav-box">
            <h3><a href="../../../community/index.html">Community</a></h3>
            <p>If you need help, you can connect with other Kubernetes users and the Kubernetes authors, attend community events, and watch video presentations from around the web.</p>
        </div>
        <div class="nav-box">
            <h3><a href="../../../blog/index.html">Blog</a></h3>
            <p>Read the latest news for Kubernetes and the containers space in general, and get technical how-tos hot off the presses.</p>
        </div>
        </main>
        <main data-auto-burger="primary">
        <div class="left">
            <h5 class="github-invite">Interested in hacking on the core Kubernetes code base?</h5>
            <a href="https://github.com/kubernetes/kubernetes" class="button" data-auto-burger-exclude>View On Github</a>
        </div>

        <div class="right">
            <h5 class="github-invite">Explore the community</h5>
            <div class="social">
                <a href="https://twitter.com/kubernetesio" class="twitter"><span>Twitter</span></a>
                <a href="https://github.com/kubernetes/kubernetes" class="github"><span>Github</span></a>
                <a href="http://slack.k8s.io/" class="slack"><span>Slack</span></a>
                <a href="http://stackoverflow.com/questions/tagged/kubernetes" class="stack-overflow"><span>Stack Overflow</span></a>
                <a href="https://discuss.kubernetes.io" class="mailing-list"><span>Forum</span></a>
                <a href="https://calendar.google.com/calendar/embed?src=nt2tcnbtbied3l6gi2h29slvc0%40group.calendar.google.com" class="calendar"><span>Events Calendar</span></a>
            </div>
        </div>
        <div class="clear" style="clear: both"></div>
        </main>
    </nav>
</header>

		
		
		<section id="hero" class="light-text no-sub">
			











<h1>Tasks</h1>
<h5></h5>






<div id="vendorStrip" class="light-text">
	<ul>
		
		
		<li><a href="../../home.1">DOCUMENTATION</a></li>
		
		
		<li><a href="../../setup/index.html">SETUP</a></li>
		
		
		<li><a href="../../concepts/index.html">CONCEPTS</a></li>
		
		
		<li><a href="../../tasks/index.html" class="YAH">TASKS</a></li>
		
		
		<li><a href="../../tutorials/index.html">TUTORIALS</a></li>
		
		
		<li><a href="../../reference.1">REFERENCE</a></li>
		
	</ul>
	<div id="searchBox">
		<input type="text" id="search" placeholder="Search" onkeydown="if (event.keyCode==13) window.location.replace('/docs/search/?q=' + this.value)" autofocus="autofocus">
	</div>
</div>

		</section>
		
		
<section id="deprecationWarning">
  <main>
    <div class="content deprecation-warning">
      <h3>
        Documentation for Kubernetes v1.11 is no longer actively maintained. The version you are currently viewing is a static snapshot.
        For up-to-date documentation, see the <a href="https://kubernetes.io/docs/home/">latest</a> version.
      </h3>
    </div>
  </main>
</section>


		<section id="encyclopedia">
			
<div id="docsToc">
     <div class="pi-accordion">
    	
        
        
        
        
        
         
             
                 
             
         
             
                 
             
         
             
                 
             
         
             
                 
             
         
             
                 
                          
                          
                 
             
         
             
         
             
         
             
         
         
        
        <a class="item" data-title="Tasks" href="../../tasks/index.html"></a>

	
	
		
		
	<div class="item" data-title="Install Tools">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Install and Set Up kubectl" href="../../tasks/kubectl/install/index.html"></a>

		
	
		
		
<a class="item" data-title="Install Minikube" href="../../tasks/tools/install-minikube/index.html"></a>

		
	
		
		
<a class="item" data-title="Installing kubeadm" href="../../setup/independent/install-kubeadm/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Configure Pods and Containers">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Assign Memory Resources to Containers and Pods" href="../../tasks/configure-pod-container/assign-cpu-ram-container"></a>

		
	
		
		
<a class="item" data-title="Assign CPU Resources to Containers and Pods" href="../../tasks/configure-pod-container/assign-cpu-resource/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Quality of Service for Pods" href="../../tasks/configure-pod-container/quality-service-pod/index.html"></a>

		
	
		
		
<a class="item" data-title="Assign Extended Resources to a Container" href="../../tasks/configure-pod-container/extended-resource/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure a Pod to Use a Volume for Storage" href="../../tasks/configure-pod-container/configure-volume-storage/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure a Pod to Use a PersistentVolume for Storage" href="../../tasks/configure-pod-container/configure-persistent-volume-storage/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure a Pod to Use a Projected Volume for Storage" href="../../tasks/configure-pod-container/configure-projected-volume-storage/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure a Security Context for a Pod or Container" href="../../user-guide/security-context"></a>

		
	
		
		
<a class="item" data-title="Configure Service Accounts for Pods" href="../../user-guide/service-accounts"></a>

		
	
		
		
<a class="item" data-title="Pull an Image from a Private Registry" href="../../tasks/configure-pod-container/pull-image-private-registry/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Liveness and Readiness Probes" href="../../user-guide/liveness/index.html"></a>

		
	
		
		
<a class="item" data-title="Assign Pods to Nodes" href="../../tasks/configure-pod-container/assign-pods-nodes/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Pod Initialization" href="../../tasks/configure-pod-container/configure-pod-initialization/index.html"></a>

		
	
		
		
<a class="item" data-title="Attach Handlers to Container Lifecycle Events" href="../../tasks/configure-pod-container/attach-handler-lifecycle-event/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure a Pod to Use a ConfigMap" href="../../tasks/configure-pod-container/configure-pod-configmap/index.html"></a>

		
	
		
		
<a class="item" data-title="Share Process Namespace between Containers in a Pod" href="../../tasks/configure-pod-container/share-process-namespace/index.html"></a>

		
	
		
		
<a class="item" data-title="Translate a Docker Compose File to Kubernetes Resources" href="../../tasks/configure-pod-container/translate-compose-kubernetes/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Administer a Cluster">
		<div class="container">
		
		
	
	
		
		
	<div class="item" data-title="Administration with kubeadm">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Upgrading kubeadm HA clusters from 1.9.x to 1.9.y" href="../../tasks/administer-cluster/kubeadm/kubeadm-upgrade-ha/index.html"></a>

		
	
		
		
<a class="item" data-title="Upgrading kubeadm clusters from 1.7 to 1.8" href="../../tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-8/index.html"></a>

		
	
		
		
<a class="item" data-title="Upgrading kubeadm clusters from v1.10 to v1.11" href="../../tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-11/index.html"></a>

		
	
		
		
<a class="item" data-title="Upgrading/downgrading kubeadm clusters between v1.8 to v1.9" href="../../tasks/administer-cluster/kubeadm/kubeadm-upgrade-1-9/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Manage Memory, CPU, and API Resources">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Configure Default Memory Requests and Limits for a Namespace" href="../../tasks/configure-pod-container/limit-range/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Default CPU Requests and Limits for a Namespace" href="../../tasks/administer-cluster/cpu-default-namespace/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Minimum and Maximum Memory Constraints for a Namespace" href="../../tasks/administer-cluster/memory-constraint-namespace/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Minimum and Maximum CPU Constraints for a Namespace" href="../../tasks/administer-cluster/cpu-constraint-namespace/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Memory and CPU Quotas for a Namespace" href="../../tasks/administer-cluster/quota-memory-cpu-namespace/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure a Pod Quota for a Namespace" href="../../tasks/administer-cluster/quota-pod-namespace/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Install a Network Policy Provider">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Use Calico for NetworkPolicy" href="../../tasks/administer-cluster/network-policy-provider/calico-network-policy/index.html"></a>

		
	
		
		
<a class="item" data-title="Use Cilium for NetworkPolicy" href="../../tasks/administer-cluster/network-policy-provider/cilium-network-policy/index.html"></a>

		
	
		
		
<a class="item" data-title="Use Kube-router for NetworkPolicy" href="../../tasks/administer-cluster/network-policy-provider/kube-router-network-policy/index.html"></a>

		
	
		
		
<a class="item" data-title="Romana for NetworkPolicy" href="../../tasks/administer-cluster/network-policy-provider/romana-network-policy/index.html"></a>

		
	
		
		
<a class="item" data-title="Weave Net for NetworkPolicy" href="../../tasks/administer-cluster/network-policy-provider/weave-network-policy/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
<a class="item" data-title="Access Clusters Using the Kubernetes API" href="../../tasks/administer-cluster/access-cluster-api/index.html"></a>

		
	
		
		
<a class="item" data-title="Access Services Running on Clusters" href="../../tasks/administer-cluster/access-cluster-services/index.html"></a>

		
	
		
		
<a class="item" data-title="Advertise Extended Resources for a Node" href="../../tasks/administer-cluster/extended-resource-node/index.html"></a>

		
	
		
		
<a class="item" data-title="Autoscale the DNS Service in a Cluster" href="../../tasks/administer-cluster/dns-horizontal-autoscaling/index.html"></a>

		
	
		
		
<a class="item" data-title="Change the Reclaim Policy of a PersistentVolume" href="../../tasks/administer-cluster/change-pv-reclaim-policy/index.html"></a>

		
	
		
		
<a class="item" data-title="Change the default StorageClass" href="../../tasks/administer-cluster/change-default-storage-class/index.html"></a>

		
	
		
		
<a class="item" data-title="Cluster Management" href="../cluster-management/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Multiple Schedulers" href="../../tasks/administer-cluster/configure-multiple-schedulers/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Out Of Resource Handling" href="../../tasks/administer-cluster/reserve-compute-resources/out-of-resource.md"></a>

		
	
		
		
<a class="item" data-title="Configure Quotas for API Objects" href="../../tasks/administer-cluster/quota-api-object/index.html"></a>

		
	
		
		
<a class="item" data-title="Control CPU Management Policies on the Node" href="../../tasks/administer-cluster/cpu-management-policies/index.html"></a>

		
	
		
		
<a class="item" data-title="Customizing DNS Service" href="../../tasks/administer-cluster/dns-custom-nameservers/index.html"></a>

		
	
		
		
<a class="item" data-title="Debugging DNS Resolution" href="../../tasks/administer-cluster/dns-debugging-resolution/index.html"></a>

		
	
		
		
<a class="item" data-title="Declare Network Policy" href="../../tasks/configure-pod-container/declare-network-policy/index.html"></a>

		
	
		
		
<a class="item" data-title="Developing Cloud Controller Manager" href="../../tasks/administer-cluster/developing-cloud-controller-manager.md"></a>

		
	
		
		
<a class="item" data-title="Encrypting Secret Data at Rest" href="../../tasks/administer-cluster/encrypt-data.1"></a>

		
	
		
		
<a class="item" data-title="Guaranteed Scheduling For Critical Add-On Pods" href="../../tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/index.html"></a>

		
	
		
		
<a class="item" data-title="IP Masquerade Agent User Guide" href="../../tasks/administer-cluster/ip-masq-agent/index.html"></a>

		
	
		
		
<a class="item" data-title="Kubernetes Cloud Controller Manager" href="../../tasks/administer-cluster/running-cloud-controller.md"></a>

		
	
		
		
<a class="item" data-title="Limit Storage Consumption" href="../../tasks/administer-cluster/limit-storage-consumption/index.html"></a>

		
	
		
		
<a class="item" data-title="Namespaces Walkthrough" href="../../tasks/administer-cluster/namespaces-walkthrough/index.html"></a>

		
	
		
		
<a class="item" data-title="Operating etcd clusters for Kubernetes" href="../../tasks/administer-cluster/configure-upgrade-etcd/index.html"></a>

		
	
		
		
<a class="item" data-title="Reconfigure a Node&#39;s Kubelet in a Live Cluster" href="../../tasks/administer-cluster/reconfigure-kubelet.1"></a>

		
	
		
		
<a class="item" data-title="Reserve Compute Resources for System Daemons" href="../../tasks/administer-cluster/reserve-compute-resources/index.html"></a>

		
	
		
		
<a class="item" data-title="Safely Drain a Node while Respecting Application SLOs" href="../../tasks/administer-cluster/safely-drain-node/index.html"></a>

		
	
		
		
<a class="item" data-title="Securing a Cluster" href="../../tasks/administer-cluster/securing-a-cluster/index.html"></a>

		
	
		
		
<a class="item" data-title="Set Kubelet parameters via a config file" href="../../tasks/administer-cluster/kubelet-config-file.1"></a>

		
	
		
		
<a class="item" data-title="Set up High-Availability Kubernetes Masters" href="../../tasks/administer-cluster/highly-available-master/index.html"></a>

		
	
		
		
<a class="item" data-title="Set up a Highly Availabile etcd Cluster With kubeadm" href="../../tasks/administer-cluster/setup-ha-etcd-with-kubeadm/index.html"></a>

		
	
		
		
<a class="item" data-title="Share a Cluster with Namespaces" href="../namespaces"></a>

		
	
		
		
<a class="item" data-title="Static Pods" href="../../concepts/cluster-administration/static-pod/index.html"></a>

		
	
		
		
<a class="item" data-title="Storage Object in Use Protection" href="../../tasks/administer-cluster/storage-object-in-use-protection/index.html"></a>

		
	
		
		
<a class="item" data-title="Using CoreDNS for Service Discovery" href="../../tasks/administer-cluster/coredns/index.html"></a>

		
	
		
		
<a class="item" data-title="Using a KMS provider for data encryption" href="../../tasks/administer-cluster/kms-provider/index.html"></a>

		
	
		
		
<a class="item" data-title="Using sysctls in a Kubernetes Cluster" href="../../concepts/cluster-administration/sysctl-cluster/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Inject Data Into Applications">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Define a Command and Arguments for a Container" href="../../user-guide/containers/index.html"></a>

		
	
		
		
<a class="item" data-title="Define Environment Variables for a Container" href="../../tasks/inject-data-application/define-environment-variable-container/index.html"></a>

		
	
		
		
<a class="item" data-title="Expose Pod Information to Containers Through Environment Variables" href="../../tasks/configure-pod-container/environment-variable-expose-pod-information/index.html"></a>

		
	
		
		
<a class="item" data-title="Expose Pod Information to Containers Through Files" href="../../tasks/inject-data-application/downward-api-volume-expose-pod-information/index.html"></a>

		
	
		
		
<a class="item" data-title="Distribute Credentials Securely Using Secrets" href="../../tasks/inject-data-application/distribute-credentials-secure/index.html"></a>

		
	
		
		
<a class="item" data-title="Inject Information into Pods Using a PodPreset" href="../../tasks/inject-data-application/podpreset.1"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Run Applications">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Run a Stateless Application Using a Deployment" href="../../user-guide/simple-nginx"></a>

		
	
		
		
<a class="item" data-title="Run a Single-Instance Stateful Application" href="../../tutorials/stateful-application/run-stateful-application/index.html"></a>

		
	
		
		
<a class="item" data-title="Run a Replicated Stateful Application" href="../../tasks/run-application/run-replicated-stateful-application/index.html"></a>

		
	
		
		
<a class="item" data-title="Update API Objects in Place Using kubectl patch" href="../../tasks/run-application/update-api-object-kubectl-patch/index.html"></a>

		
	
		
		
<a class="item" data-title="Scale a StatefulSet" href="../../tasks/run-application/scale-stateful-set/index.html"></a>

		
	
		
		
<a class="item" data-title="Delete a StatefulSet" href="../../tasks/manage-stateful-set/delete-pods/index.html"></a>

		
	
		
		
<a class="item" data-title="Force Delete StatefulSet Pods" href="../../tasks/run-application/force-delete-stateful-set-pod/index.html"></a>

		
	
		
		
<a class="item" data-title="Perform Rolling Update Using a Replication Controller" href="../../tasks/run-application/rolling-update-replication-controller/index.html"></a>

		
	
		
		
<a class="item" data-title="Horizontal Pod Autoscaler" href="../../tasks/run-application/horizontal-pod-autoscale/index.html"></a>

		
	
		
		
<a class="item" data-title="Horizontal Pod Autoscaler Walkthrough" href="../../tasks/run-application/horizontal-pod-autoscale-walkthrough/index.html"></a>

		
	
		
		
<a class="item" data-title="Specifying a Disruption Budget for your Application" href="../../tasks/configure-pod-container/configure-pod-disruption-budget/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Run Jobs">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Running automated tasks with cron jobs" href="../../tasks/job/automated-tasks-with-cron-jobs.1"></a>

		
	
		
		
<a class="item" data-title="Parallel Processing using Expansions" href="../../tasks/job/parallel-processing-expansion/index.html"></a>

		
	
		
		
<a class="item" data-title="Coarse Parallel Processing Using a Work Queue" href="../../tasks/job/coarse-parallel-processing-work-queue/index.html"></a>

		
	
		
		
<a class="item" data-title="Fine Parallel Processing Using a Work Queue" href="../../tasks/job/fine-parallel-processing-work-queue/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Access Applications in a Cluster">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Web UI (Dashboard)" href="../../tasks/web-ui-dashboard/index.html"></a>

		
	
		
		
<a class="item" data-title="Accessing Clusters" href="../../concepts/cluster-administration/access-cluster/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure Access to Multiple Clusters" href="../../tasks/access-application-cluster/authenticate-across-clusters-kubeconfig/index.html"></a>

		
	
		
		
<a class="item" data-title="Use Port Forwarding to Access Applications in a Cluster" href="../../tasks/access-application-cluster/port-forward-access-application-cluster/index.html"></a>

		
	
		
		
<a class="item" data-title="Provide Load-Balanced Access to an Application in a Cluster" href="../../tasks/access-application-cluster/load-balance-access-application-cluster/index.html"></a>

		
	
		
		
<a class="item" data-title="Use a Service to Access an Application in a Cluster" href="../../tasks/access-application-cluster/service-access-application-cluster.1"></a>

		
	
		
		
<a class="item" data-title="Connect a Front End to a Back End Using a Service" href="../../tasks/access-application-cluster/connecting-frontend-backend/index.html"></a>

		
	
		
		
<a class="item" data-title="Create an External Load Balancer" href="../../user-guide/load-balancer"></a>

		
	
		
		
<a class="item" data-title="Configure Your Cloud Provider&#39;s Firewalls" href="../../tasks/access-application-cluster/configure-cloud-provider-firewall/index.html"></a>

		
	
		
		
<a class="item" data-title="List All Container Images Running in a Cluster" href="../../tasks/access-application-cluster/list-all-running-container-images/index.html"></a>

		
	
		
		
<a class="item" data-title="Communicate Between Containers in the Same Pod Using a Shared Volume" href="../../tasks/access-application-cluster/communicate-containers-same-pod-shared-volume/index.html"></a>

		
	
		
		
<a class="item" data-title="Configure DNS for a Cluster" href="../../tasks/access-application-cluster/configure-dns-cluster/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Monitor, Log, and Debug">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Application Introspection and Debugging" href="../../tasks/debug-application-cluster/debug-application-introspection/index.html"></a>

		
	
		
		
<a class="item" data-title="Auditing" href="../../tasks/debug-application-cluster/audit/index.html"></a>

		
	
		
		
<a class="item" data-title="Core metrics pipeline" href="../../tasks/debug-application-cluster/core-metrics-pipeline/index.html"></a>

		
	
		
		
<a class="item" data-title="Debug Init Containers" href="../../tasks/debug-application-cluster/debug-init-containers/index.html"></a>

		
	
		
		
<a class="item" data-title="Debug Pods and Replication Controllers" href="../../tasks/debug-application-cluster/debug-pod-replication-controller/index.html"></a>

		
	
		
		
<a class="item" data-title="Debug Services" href="../../user-guide/debugging-services"></a>

		
	
		
		
<a class="item" data-title="Debug a StatefulSet" href="../../tasks/manage-stateful-set/debugging-a-statefulset/index.html"></a>

		
	
		
		
<a class="item" data-title="Debugging Kubernetes nodes with crictl" href="../../tasks/debug-application-cluster/crictl/index.html"></a>

		
	
		
		
<a class="item" data-title="Determine the Reason for Pod Failure" href="../../tasks/debug-application-cluster/determine-reason-pod-failure/index.html"></a>

		
	
		
		
<a class="item" data-title="Developing and debugging services locally" href="../../tasks/debug-application-cluster/local-debugging/index.html"></a>

		
	
		
		
<a class="item" data-title="Events in Stackdriver" href="../../tasks/debug-application-cluster/events-stackdriver/index.html"></a>

		
	
		
		
<a class="item" data-title="Get a Shell to a Running Container" href="../../tasks/debug-application-cluster/get-shell-running-container/index.html"></a>

		
	
		
		
<a class="item" data-title="Logging Using Elasticsearch and Kibana" href="../../user-guide/logging/elasticsearch.1"></a>

		
	
		
		
<a class="item" data-title="Logging Using Stackdriver" href="../../user-guide/logging/stackdriver.1"></a>

		
	
		
		
<a class="item" data-title="Monitor Node Health" href="../../tasks/debug-application-cluster/monitor-node-health/index.html"></a>

		
	
		
		
<a class="item" data-title="Tools for Monitoring Compute, Storage, and Network Resources" href="../../tasks/debug-application-cluster/resource-usage-monitoring/index.html"></a>

		
	
		
		
<a class="item" data-title="Troubleshoot Applications" href="../../tasks/debug-application-cluster/debug-application.1"></a>

		
	
		
		
<a class="item" data-title="Troubleshoot Clusters" href="../cluster-troubleshooting.1"></a>

		
	
		
		
<a class="item" data-title="Troubleshooting" href="../../troubleshooting/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Extend Kubernetes">
		<div class="container">
		
		
	
	
		
		
	<div class="item" data-title="Use Custom Resources">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Extend the Kubernetes API with CustomResourceDefinitions" href="../../tasks/access-kubernetes-api/extend-api-custom-resource-definitions/index.html"></a>

		
	
		
		
<a class="item" data-title="Versions of CustomResourceDefinitions" href="../../tasks/access-kubernetes-api/custom-resources/custom-resource-definition-versioning/index.html"></a>

		
	
		
		
<a class="item" data-title="Migrate a ThirdPartyResource to CustomResourceDefinition" href="../../tasks/access-kubernetes-api/migrate-third-party-resource/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
<a class="item" data-title="Configure the aggregation layer" href="../../tasks/access-kubernetes-api/configure-aggregation-layer/index.html"></a>

		
	
		
		
<a class="item" data-title="Setup an extension API server" href="../../tasks/access-kubernetes-api/setup-extension-api-server/index.html"></a>

		
	
		
		
<a class="item" data-title="Use an HTTP Proxy to Access the Kubernetes API" href="../../tasks/access-kubernetes-api/http-proxy-access-api.1"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="TLS">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Certificate Rotation" href="../../tasks/tls/certificate-rotation/index.html"></a>

		
	
		
		
<a class="item" data-title="Manage TLS Certificates in a Cluster" href="../../tasks/tls/managing-tls-in-a-cluster.1"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Federation - Run an App on Multiple Clusters">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Cross-cluster Service Discovery using Federated Services" href="../../concepts/cluster-administration/federation-service-discovery/index.html"></a>

		
	
		
		
<a class="item" data-title="Set up Cluster Federation with Kubefed" href="../../tutorials/federation/set-up-cluster-federation-kubefed/index.html"></a>

		
	
		
		
<a class="item" data-title="Set up CoreDNS as DNS provider for Cluster Federation" href="../../tasks/federation/set-up-coredns-provider-federation/index.html"></a>

		
	
		
		
<a class="item" data-title="Set up placement policies in Federation" href="../../tasks/federation/set-up-placement-policies-federation/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Manage Cluster Daemons">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Perform a Rolling Update on a DaemonSet" href="../../tasks/manage-daemon/update-daemon-set/index.html"></a>

		
	
		
		
<a class="item" data-title="Performing a Rollback on a DaemonSet" href="../../tasks/manage-daemon/rollback-daemon-set/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Install Service Catalog">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Install Service Catalog using Helm" href="../../tasks/service-catalog/install-service-catalog-using-helm/index.html"></a>

		
	
		
		
<a class="item" data-title="Install Service Catalog using SC" href="../../tasks/service-catalog/install-service-catalog-using-sc/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Federation - Run an App on Multiple Clusters">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Federated Cluster" href="../../tasks/administer-federation/cluster/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated ConfigMap" href="../../tasks/administer-federation/configmap/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated DaemonSet" href="../../tasks/administer-federation/daemonset/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Deployment" href="../../tasks/administer-federation/deployment/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Events" href="../../tasks/administer-federation/events/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Horizontal Pod Autoscalers (HPA)" href="../../tasks/administer-federation/hpa/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Ingress" href="../../tasks/administer-federation/ingress/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Jobs" href="../../tasks/administer-federation/job/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Namespaces" href="../../tasks/administer-federation/namespaces/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated ReplicaSets" href="../../tasks/administer-federation/replicaset/index.html"></a>

		
	
		
		
<a class="item" data-title="Federated Secrets" href="../../tasks/administer-federation/secret/index.html"></a>

		
	

		</div>
	</div>

		
	
		
		
<a class="item" data-title="Extend kubectl with plugins" href="../../tasks/extend-kubectl/kubectl-plugins/index.html"></a>

		
	
		
		
<a class="item" data-title="Manage HugePages" href="../../tasks/manage-hugepages/scheduling-hugepages/index.html"></a>

		
	
		
		
<a class="item" data-title="Schedule GPUs" href="../../tasks/manage-gpus/scheduling-gpus/index.html"></a>

		
	






     </div> 
    <button class="push-menu-close-button" onclick="kub.toggleToc()"></button>
</div> 

			<div id="docsContent">
				
<p><a href="../../editdocs#docs/tasks/debug-application-cluster/debug-cluster.md" id="editPageButton">Edit This Page</a></p>

<h1>Troubleshoot Clusters</h1>



<p>This doc is about cluster troubleshooting; we assume you have already ruled out your application as the root cause of the
problem you are experiencing. See
the <a href="../../tasks/debug-application-cluster/debug-application.1">application troubleshooting guide</a> for tips on application debugging.
You may also visit <a href="../../troubleshooting/index.html">troubleshooting document</a> for more information.</p>









<ul id="markdown-toc">










<li><a href="../cluster-troubleshooting.1#listing-your-cluster">Listing your cluster</a></li>




<li><a href="../cluster-troubleshooting.1#looking-at-logs">Looking at logs</a></li>




<li><a href="../cluster-troubleshooting.1#a-general-overview-of-cluster-failure-modes">A general overview of cluster failure modes</a></li>



















</ul>


<h2 id="listing-your-cluster">Listing your cluster</h2>

<p>The first thing to debug in your cluster is if your nodes are all registered correctly.</p>

<p>Run</p>
<div class="highlight"><pre style="background-color:#f8f8f8;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-shell" data-lang="shell">kubectl get nodes</code></pre></div>
<p>And verify that all of the nodes you expect to see are present and that they are all in the <code>Ready</code> state.</p>

<h2 id="looking-at-logs">Looking at logs</h2>

<p>For now, digging deeper into the cluster requires logging into the relevant machines.  Here are the locations
of the relevant log files.  (note that on systemd-based systems, you may need to use <code>journalctl</code> instead)</p>

<h3 id="master">Master</h3>

<ul>
<li>/var/log/kube-apiserver.log - API Server, responsible for serving the API</li>
<li>/var/log/kube-scheduler.log - Scheduler, responsible for making scheduling decisions</li>
<li>/var/log/kube-controller-manager.log - Controller that manages replication controllers</li>
</ul>

<h3 id="worker-nodes">Worker Nodes</h3>

<ul>
<li>/var/log/kubelet.log - Kubelet, responsible for running containers on the node</li>
<li>/var/log/kube-proxy.log - Kube Proxy, responsible for service load balancing</li>
</ul>

<h2 id="a-general-overview-of-cluster-failure-modes">A general overview of cluster failure modes</h2>

<p>This is an incomplete list of things that could go wrong, and how to adjust your cluster setup to mitigate the problems.</p>

<p>Root causes:</p>

<ul>
<li>VM(s) shutdown</li>
<li>Network partition within cluster, or between cluster and users</li>
<li>Crashes in Kubernetes software</li>
<li>Data loss or unavailability of persistent storage (e.g. GCE PD or AWS EBS volume)</li>
<li>Operator error, e.g. misconfigured Kubernetes software or application software</li>
</ul>

<p>Specific scenarios:</p>

<ul>
<li>Apiserver VM shutdown or apiserver crashing

<ul>
<li>Results

<ul>
<li>unable to stop, update, or start new pods, services, replication controller</li>
<li>existing pods and services should continue to work normally, unless they depend on the Kubernetes API</li>
</ul></li>
</ul></li>
<li>Apiserver backing storage lost

<ul>
<li>Results

<ul>
<li>apiserver should fail to come up</li>
<li>kubelets will not be able to reach it but will continue to run the same pods and provide the same service proxying</li>
<li>manual recovery or recreation of apiserver state necessary before apiserver is restarted</li>
</ul></li>
</ul></li>
<li>Supporting services (node controller, replication controller manager, scheduler, etc) VM shutdown or crashes

<ul>
<li>currently those are colocated with the apiserver, and their unavailability has similar consequences as apiserver</li>
<li>in future, these will be replicated as well and may not be co-located</li>
<li>they do not have their own persistent state</li>
</ul></li>
<li>Individual node (VM or physical machine) shuts down

<ul>
<li>Results

<ul>
<li>pods on that Node stop running</li>
</ul></li>
</ul></li>
<li>Network partition

<ul>
<li>Results

<ul>
<li>partition A thinks the nodes in partition B are down; partition B thinks the apiserver is down. (Assuming the master VM ends up in partition A.)</li>
</ul></li>
</ul></li>
<li>Kubelet software fault

<ul>
<li>Results

<ul>
<li>crashing kubelet cannot start new pods on the node</li>
<li>kubelet might delete the pods or not</li>
<li>node marked unhealthy</li>
<li>replication controllers start new pods elsewhere</li>
</ul></li>
</ul></li>
<li>Cluster operator error

<ul>
<li>Results

<ul>
<li>loss of pods, services, etc</li>
<li>lost of apiserver backing store</li>
<li>users unable to read API</li>
<li>etc.</li>
</ul></li>
</ul></li>
</ul>

<p>Mitigations:</p>

<ul>
<li><p>Action: Use IaaS provider&rsquo;s automatic VM restarting feature for IaaS VMs</p>

<ul>
<li>Mitigates: Apiserver VM shutdown or apiserver crashing</li>
<li>Mitigates: Supporting services VM shutdown or crashes</li>
</ul></li>

<li><p>Action: Use IaaS providers reliable storage (e.g. GCE PD or AWS EBS volume) for VMs with apiserver+etcd</p>

<ul>
<li>Mitigates: Apiserver backing storage lost</li>
</ul></li>

<li><p>Action: Use (experimental) <a href="../high-availability">high-availability</a> configuration</p>

<ul>
<li>Mitigates: Master VM shutdown or master components (scheduler, API server, controller-managing) crashing</li>
<li>Will tolerate one or more simultaneous node or component failures</li>
<li>Mitigates: Apiserver backing storage (i.e., etcd&rsquo;s data directory) lost</li>
<li>Assuming you used clustered etcd.</li>
</ul></li>

<li><p>Action: Snapshot apiserver PDs/EBS-volumes periodically</p>

<ul>
<li>Mitigates: Apiserver backing storage lost</li>
<li>Mitigates: Some cases of operator error</li>
<li>Mitigates: Some cases of Kubernetes software fault</li>
</ul></li>

<li><p>Action: use replication controller and services in front of pods</p>

<ul>
<li>Mitigates: Node shutdown</li>
<li>Mitigates: Kubelet software fault</li>
</ul></li>

<li><p>Action: applications (containers) designed to tolerate unexpected restarts</p>

<ul>
<li>Mitigates: Node shutdown</li>
<li>Mitigates: Kubelet software fault</li>
</ul></li>

<li><p>Action: <a href="../../concepts/cluster-administration/federation/index.html">Multiple independent clusters</a> (and avoid making risky changes to all clusters at once)</p>

<ul>
<li>Mitigates: Everything listed above.</li>
</ul></li>
</ul>
















				<div class="issue-button-container">
					<p><a href="../cluster-troubleshooting.1"><img src="https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/tasks/debug-application-cluster/debug-cluster.md?pixel" alt="Analytics" /></a></p>
					
					
					<script type="text/javascript">
					PDRTJS_settings_8345992 = {
					"id" : "8345992",
					"unique_id" : "\/docs\/tasks\/debug-application-cluster\/debug-cluster\/",
					"title" : "Troubleshoot Clusters",
					"permalink" : "https:\/\/kubernetes.io\/docs\/tasks\/debug-application-cluster\/debug-cluster\/"
					};
					(function(d,c,j){if(!document.getElementById(j)){var pd=d.createElement(c),s;pd.id=j;pd.src=('https:'==document.location.protocol)?'https://polldaddy.com/js/rating/rating.js':'http://i0.poll.fm/js/rating/rating.js';s=document.getElementsByTagName(c)[0];s.parentNode.insertBefore(pd,s);}}(document,'script','pd-rating-js'));
					</script>
					<a href="../cluster-troubleshooting.1" onclick="window.open('https://github.com/kubernetes/website/issues/new?title=Issue%20with%20' +
					'k8s.io'+window.location.pathname)" class="button issue">Create an Issue</a>
					
					
					
					<a href="../../editdocs#docs/tasks/debug-application-cluster/debug-cluster.md" class="button issue">Edit this Page</a>
					
				</div>
			</div>
		</section>
		<footer>
    <main class="light-text">
        <nav>
            
            
            
            <a href="../../home.1">Documentation</a>
            
            <a href="../../../blog/index.html">Blog</a>
            
            <a href="../../../partners/index.html">Partners</a>
            
            <a href="../../../community/index.html">Community</a>
            
            <a href="../../../case-studies/index.html">Case Studies</a>
            
        </nav>
        <div class="social">
            <div>
                <a href="https://twitter.com/kubernetesio" class="twitter"><span>twitter</span></a>
                <a href="https://github.com/kubernetes/kubernetes" class="github"><span>Github</span></a>
                <a href="http://slack.k8s.io/" class="slack"><span>Slack</span></a>
            </div>
            <div>
                <a href="http://stackoverflow.com/questions/tagged/kubernetes" class="stack-overflow"><span>Stack Overflow</span></a>
                <a href="https://discuss.kubernetes.io" class="mailing-list"><span>Forum</span></a>
                <a href="https://calendar.google.com/calendar/embed?src=nt2tcnbtbied3l6gi2h29slvc0%40group.calendar.google.com" class="calendar"><span>Events Calendar</span></a>
            </div>
            <div>
                <a href="../../getting-started-guides/index.html" class="button">Get Kubernetes</a>
                <a href="https://git.k8s.io/community/contributors/guide" class="button">Contribute</a>
            </div>
        </div>
        <div id="miceType" class="center">
            &copy; 2018 The Kubernetes Authors | Documentation Distributed under <a href="https://git.k8s.io/website/LICENSE" class="light-text">CC BY 4.0</a>
        </div>
        <div id="miceType" class="center">
            Copyright &copy; 2018 The Linux Foundation&reg;. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our <a href="https://www.linuxfoundation.org/trademark-usage" class="light-text">Trademark Usage page</a>
        </div>
    </main>
</footer>

		<button class="flyout-button" onclick="kub.toggleToc()"></button>

<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
    (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-36037335-10', 'auto');
ga('send', 'pageview');


(function () {
    window.addEventListener('DOMContentLoaded', init)

        
        function init() {
            window.removeEventListener('DOMContentLoaded', init)
                hideNav()
        }

    function hideNav(toc){
        if (!toc) toc = document.querySelector('#docsToc')
        if (!toc) return
            var container = toc.querySelector('.container')

                
                if (container) {
                    if (container.childElementCount === 0 || toc.querySelectorAll('a.item').length === 1) {
                        toc.style.display = 'none'
                            document.getElementById('docsContent').style.width = '100%'
                    }
                } else {
                    requestAnimationFrame(function () {
                        hideNav(toc)
                    })
                }
    }
})();
</script>



	</body>
</html>